docs(babysit-pipeline): add gap-backfill mode and the traps it exposed by MarkusNeusinger · Pull Request #10628 · MarkusNeusinger/anyplot

MarkusNeusinger · 2026-08-24T20:41:35Z

Why

The babysit-pipeline skill covers watching one fresh spec to 15/15. Backfilling coverage gaps across the catalogue — dozens of older specs each missing the libraries added after they were generated — is a different job, and the skill said nothing about it. Running it by hand on 2026-08-24 surfaced four traps that cost, or nearly cost, real implementations.

What's new

Section 5 · Gap backfill

Compute the missing set from plots/{spec}/metadata/{lang}/{lib}.yaml on origin/main, not from impl:{lib}:done labels — most affected specs have closed issues, so their labels are absent or stale. Includes the git ls-tree | awk one-liner.
run_spec.sh <spec> <model> <lib>... as the driver: staggered dispatch, poll to metadata, RESULT=COMPLETE|PARTIAL|TIMEOUT, skips libraries already on main so a re-run retries exactly the gaps.
Two specs in parallel (~4 specs/h vs ~2; ten concurrent impl-generate runs showed no rate-limit effects).
A done.log / deferred.log ledger written before dispatching the next spec, so a compaction or crashed session resumes without recounting.
The one-retry rule, with the evidence for why it is not optional: the single retry recovered highcharts/treemap-basic, ggplot2/wireframe-3d-basic and ggplot2/network-force-directed, all three of which had already been read as capability gaps.

Four new gotchas

Trap	Why it matters
`Marking <lib> as failed: N generation attempts` counts more than this run	The markers span past campaigns (#10627 scopes this to 12 h). Check the `Previous failures for <lib>/<spec>: N` notice before concluding a library can't do a plot type.
`impl:<lib>:failed` is terminal and unreliable	Nothing re-dispatches it — watchdog case 3 fires once, then only logs `needs manual attention`. And of 87 such labels, 42 sat on implementations that had since landed.
"Agent reports success, writes no file"	8 of 85 generate runs (~9%). Self-heals when retry budget remains (`bubble-basic/highcharts`: failed 17:55, succeeded on auto-retry 18:02). One occurrence is noise.
Static library + interactive/3D spec	The one gap shape that usually is genuine — 18 of the 45 real gaps. Still give the one retry, then defer.

run_spec.sh joins the bundled scripts. Its hardcoded REPO=/home/tirao/anyplot is replaced by resolution from the script's own location (ANYPLOT_REPO overrides), matching how poll_spec.sh derives HERE.

Verification

bash -n on run_spec.sh; repo auto-resolution checked from the skill directory (git -C "$HERE" rev-parse --show-toplevel → the repo root).
The skill's existing ALL_LIBS gotcha now names run_spec.sh's lang_of too, so a library addition updates both scripts.
Content is drawn from a real backfill run, not invented: every number above has a corresponding workflow run or label query behind it.

Backfilling coverage gaps across the catalogue is a different job from babysitting one fresh spec, and the skill only covered the latter. New section 5 documents it: compute the missing set from plots/{spec}/metadata/ rather than from impl:{lib}:done labels (most of those specs have closed issues, so the labels are absent or stale), drive one spec per invocation with run_spec.sh, run two specs in parallel, keep a done/deferred ledger under agentic/runs/, and give every missing library exactly one retry before deferring it. Four gotchas from the 2026-08-24 backfill, each of which cost or nearly cost real implementations: - `Marking <lib> as failed: N generation attempts` counts markers from more than the current run; check the `Previous failures` notice before reading it as a capability gap. - `impl:<lib>:failed` is terminal (nothing re-dispatches it) and stale — 42 of 87 such labels sat on implementations that had since landed. - "Agent reports success, writes no file" is an intermittent ~9% failure that self-heals when retry budget remains; one occurrence means nothing. - Static library against an interactive or 3D spec is the gap shape that usually is genuine. run_spec.sh joins the bundled scripts, with its hardcoded repo path replaced by resolution from the script's own location (ANYPLOT_REPO overrides) so it works from any checkout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

Copilot

🟡 Changes recommended

run_spec.sh currently isn’t reliably CWD-independent as documented and has a fragile timestamp comparison in its failure counter, which can lead to incorrect idle/health reporting during backfill runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the .claude/skills/babysit-pipeline operational documentation to cover “gap backfill” (many specs with partial coverage) and adds a new per-spec driver script (run_spec.sh) to dispatch missing libraries and poll origin/main metadata until coverage lands.

Changes:

Document a new “Gap backfill” mode, including how to compute missing (spec, lib) pairs from plots/*/metadata/** on origin/main and how to pace parallel dispatch.
Add .claude/skills/babysit-pipeline/run_spec.sh to dispatch impl-generate per missing library and poll for metadata completion.
Add a changelog entry under [Unreleased] describing the new skill capabilities and gotchas.

File summaries

File	Description
`CHANGELOG.md`	Adds an `[Unreleased]` entry describing the new babysit-pipeline gap-backfill workflow and gotchas.
`.claude/skills/babysit-pipeline/SKILL.md`	Documents the new “Gap backfill” workflow and updates gotchas to reference the new driver behavior and maintenance requirements.
`.claude/skills/babysit-pipeline/run_spec.sh`	Introduces a per-spec backfill driver that dispatches missing libraries and polls `origin/main` metadata as the durable completion signal.

Review details

Suppressed comments (3)

.claude/skills/babysit-pipeline/run_spec.sh:40

pipeline_active() calls gh run list from the caller’s current working directory, even though the script claims it works from any working directory. If the user runs this script outside the repo (or without GH_REPO configured), gh will fail and the function will always assume "active", preventing idle detection from working as intended. Run gh from $REPO so the repository context is correct.

pipeline_active() {
  for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do
    local out
    if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \
        --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then

.claude/skills/babysit-pipeline/run_spec.sh:53

recent_generate_failures() compares ISO timestamps using a string comparison against todate, which can be format-sensitive and produce incorrect counts. It also runs gh from the caller’s CWD, which breaks if the script is invoked outside the repo. Compare timestamps numerically via fromdateiso8601 and run gh from $REPO.

recent_generate_failures() {
  gh run list --workflow=impl-generate.yml --limit 15 \
    --json conclusion,updatedAt \
    --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null || echo 0
}

.claude/skills/babysit-pipeline/run_spec.sh:74

The dispatch path (gh workflow run ...) is also executed from the caller’s current working directory. If the script is launched from outside the repo, dispatch will fail even though $REPO was resolved successfully. Run the gh invocation from $REPO to make the driver actually CWD-independent as documented.

  if ! gh workflow run impl-generate.yml \
      -f "specification_id=$SPEC" -f "library=$lib" -f "model=$MODEL" >> "$LOG" 2>&1; then
    echo "WARN: dispatch failed for $SPEC/$lib" | tee -a "$LOG"

Files reviewed: 3/3 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

MarkusNeusinger · 2026-08-24T20:47:51Z

+set -uo pipefail
+SPEC="$1"; MODEL="$2"; shift 2; LIBS=("$@")
+# Resolve the repo from this script's location (.claude/skills/<name>/), so the
+# driver works from any checkout and any working directory. ANYPLOT_REPO wins
+# when the script is copied elsewhere (e.g. a scratch queue under agentic/runs/).
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || echo "$PWD")}"
+STAGGER=150      # seconds between dispatches


Both applied. run_spec.sh now has an explicit arity guard ([ "$#" -ge 3 ] || usage) rather than letting set -u surface a missing argument from inside the polling loop, and repo resolution fails fast instead of falling back to $PWD — an unvalidated root makes every meta_present check return false, which reads as "nothing ever landed" and burns the full timeout before anyone notices. The error names ANYPLOT_REPO as the override. Verified: no-args exits 2 with usage, ANYPLOT_REPO=/nonexistent exits 2 with the resolution error.

Related, same failure direction: the generate-failure counter used --limit 15, which no longer covers its own 25-minute window once two specs are in flight, and masked API errors as 0. Both silently report a healthy pipeline; raised to 60 and it now reports ? on error.

MarkusNeusinger · 2026-08-24T20:47:52Z

+**Two specs in parallel is fine** (~4 specs/h vs ~2); ten concurrent
+`impl-generate` runs showed no rate-limit effects. Keep a ledger —
+`done.log` / `deferred.log` next to the queue file in `agentic/runs/`
+— and append the result line before dispatching the next spec, so a
+compaction or a crashed session can resume without recounting.


You are right, and the contradiction was load-bearing rather than cosmetic. Parallelism is safe only for run_spec.sh, which decides per library from origin/main metadata; poll_spec.sh and monitor_spec.sh infer progress from active impl-* runs, so a second spec in flight makes them call a stalled spec healthy — exactly backwards.

Both places now say so: §0 marks backfill as the single exception with the reason, and §5 states that the two modes must not be mixed on one queue.

…m claim Three points from the Copilot review on #10628: - run_spec.sh took its arguments under `set -u` with no arity check, so a missing library surfaced as an unbound-variable error from inside the polling loop. Explicit usage guard instead. - Repo resolution fell back to $PWD. An unvalidated root makes every meta_present check return false, which reads as "nothing ever landed" and burns the full timeout; now it fails fast and names ANYPLOT_REPO. - "Two specs in parallel is fine" contradicted §0 and §3. It holds only for run_spec.sh, which reads completion per library from origin/main metadata; poll_spec.sh and monitor_spec.sh infer progress from active runs and would call a stalled spec healthy with a second spec in flight. Both sections now say so. Also raise the generate-failure counter's run limit from 15 to 60 and report `?` instead of 0 on API error: 15 no longer covers the 25-minute window once two specs are in flight, and both defects fail in the same direction — silently reporting a healthy pipeline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

Copilot

🟡 Changes recommended

run_spec.sh has correctness issues around gh repo targeting and unchecked git fetch failures that can lead to dispatching/polling against the wrong repo or stale origin/main state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.claude/skills/babysit-pipeline/run_spec.sh:109

Inside the polling loop, git -C "$REPO" fetch origin main --quiet is also unchecked. A transient fetch failure would silently keep checking stale origin/main metadata and can lead to a misleading RESULT=TIMEOUT/RESULT=PARTIAL even if metadata landed.

Handle fetch errors explicitly (same as the initial fetch) so polling results reflect reality.

  git -C "$REPO" fetch origin main --quiet

Files reviewed: 3/3 changed files
Comments generated: 2
Review effort level: Lite

+REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || true)}"
+if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then
+  echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $HERE)." >&2
+  echo "       set ANYPLOT_REPO=/path/to/anyplot and re-run." >&2
+  exit 2
+fi
+STAGGER=150      # seconds between dispatches


+    || echo "?"
+}
+
+git -C "$REPO" fetch origin main --quiet


…er's fetch (#10660) ## Why #10628 documented gap backfill while the backfill was still running. Finishing it falsified three of its claims, so this PR corrects them rather than leaving confident wrong guidance in a skill. Running the backfill in two parallel slots also exposed a race in the driver, fixed here. **Scope: prose plus one executable change.** `SKILL.md` and `CHANGELOG.md` are documentation; `run_spec.sh` gains a fetch retry (behaviour change, details below). ## Corrections **1. Halt-on-cluster counted the wrong thing.** The threshold was "≥5 failed `Generate:` runs in minutes = quota exhausted". Since #10627 restored the full retry budget, a single impossible pair spends three runs on its own auto-retries — so two bad pairs trip a raw count of five while the pipeline is perfectly healthy. That happened during the run: 6 failures at 22:29–22:37, all of them `plotnine` on two specs, nothing wrong with the pipeline. Now counts distinct `(spec, library)` pairs. **2. "Static library + interactive/3D spec is the one gap that is usually real" was backwards.** Every category-level prediction made during the backfill turned out wrong: | Prediction | Outcome | |---|---| | chartjs can't do treemap / sankey | chartjs succeeded on `network-force-directed`, `arc-basic`, and every JS-block spec | | plotnine can't do 3D | `bar-3d-categorical` succeeded; `scatter-3d` did not | | ggplot2 can't do wireframe | succeeded on retry | | pygal is a lost cause (6 failed labels) | succeeded on `map-marker-clustered` and `line-stress-strain` | 17 of 20 parked pairs generated fine. Exactly three failed under a full budget — plotnine on `scatter-3d`, `contour-3d`, `line-3d-trajectory`, all of which need a spatial projection plotnine does not have, while the "3D" spec representable in 2D went through. The gotcha now says: measure, don't predict. **3. The one-retry rule needed a precondition.** The workflow now spends three attempts per campaign by itself, so a pair that comes back missing may already be measured and the manual retry adds nothing. The skill now shows the actual command — `gh run list` has no per-spec filter, so it filters the output by run title with jq. Three failures minutes apart is a gap, one is a flake. The same check resolves `RESULT=TIMEOUT` with `recent generate failures: 0`, which only means the failures aged out of the driver's 25-minute window (seen on `line-3d-trajectory`, which had in fact failed three times 40 minutes earlier). ## New gotcha Spec IDs harvested from `impl:*:failed` issue titles must be intersected with the real `plots/` directories: **14 of 26** pointed at specs that no longer exist on main, and such a dispatch dies seconds in at `Validate specification exists`. This is how the rescue list first read as 45 missing implementations when only 20 were real. ## Behaviour change: `run_spec.sh` retries its fetch Two drivers polling the same checkout collide on the ref lock: ``` error: cannot lock ref 'refs/remotes/origin/main': is at 6b666c1 but expected c736a84 ``` A lost fetch leaves `origin/main` stale, so `meta_present` understates what has landed and the poller reports `PARTIAL` for libraries that are already merged — a failure that disguises itself as a stalled spec. `fetch_main()` now retries three times with backoff and, when all three lose, logs the warning **with git's own stderr** so a ref-lock collision is distinguishable from an auth or network failure. ## Verification - `bash -n` on `run_spec.sh`; the failure path exercised against a bad ref → `last error: fatal: couldn't find remote ref …`, confirming the message survives to the log. - Every number in the corrections traces to a workflow run or label query from the 2026-08-24 backfill; the ledger is in `agentic/runs/babysit-2026-08-20/`. ## Related - #10627 (retry cap) — verified in production: `Previous failures for plotnine/contour-3d since 2026-08-24T10:38:22Z: 2` followed by `3 failed attempt(s) in the last 12h (cap: 3 per campaign)`. - #10628 — the section this corrects. - #10540 — verified by dry run: `daily-regen liveness: workflow state is 'disabled_manually', not active — rescue skipped`, scan completing successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 24, 2026 20:41

Copilot started reviewing on behalf of MarkusNeusinger August 24, 2026 20:42 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 24, 2026 20:47

Copilot started reviewing on behalf of MarkusNeusinger August 24, 2026 20:48 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

MarkusNeusinger merged commit f2c1123 into main Aug 24, 2026
11 checks passed

MarkusNeusinger deleted the docs/babysit-skill-backfill-mode branch August 24, 2026 20:51

MarkusNeusinger mentioned this pull request Aug 25, 2026

docs(babysit-pipeline): correct the backfill guidance, retry the driver's fetch #10660

Merged

Navigation Menu

docs(babysit-pipeline): add gap-backfill mode and the traps it exposed - #10628

docs(babysit-pipeline): add gap-backfill mode and the traps it exposed#10628
MarkusNeusinger merged 2 commits into
mainfrom
docs/babysit-skill-backfill-mode

MarkusNeusinger commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

MarkusNeusinger Aug 24, 2026

Uh oh!

MarkusNeusinger Aug 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MarkusNeusinger commented Aug 24, 2026

Why

What's new

Verification

Related

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (3)

Uh oh!

MarkusNeusinger Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

MarkusNeusinger Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants